Three-Dimensional Holographic Visual and Haptic Object Warning based on Visual Recognition Analysis

ABSTRACT

Presenting a haptic hologram warning is provided. An indication that a first individual who needs supervision is approaching a situation is received. A haptic hologram is presented to the first individual who needs supervision prior to the first individual reaching the situation.

BACKGROUND 1. Field

The disclosure relates generally to holograms and more specifically togenerating a three-dimensional holographic visual and haptic objectwarning based on visual recognition analysis of image data correspondingto an area surrounding a set of one or more monitored objects.

2. Description of the Related Art

The word haptic refers to the sense of touch. Haptic feedback is the useof the sense of touch in a user interface design to provide informationto a user. Typically, haptic feedback stimulates a person's sense oftouch, generally created by actuators or motors that generate avibratory sensation in the hand of the person. For example, a rumblepack for a game controller provides haptic feedback through a user'shands.

Emerging technology is making use of projected ultrasound to generatethree-dimensional (3D) objects that can be seen and felt in mid-air.This tactile sensation works by using forced ultrasound from an array ofultrasound transducers to generate and focus patterns of ultrasound toshape the air at which it was directed. These patterns of forcedultrasound induce a shear wave in skin tissue, creating a displacement,which triggers mechanoreceptors in the skin similar to feeling a solidobject. Additionally, these air shapes can be made visible by directingthe manipulated air through a thin layer of fluid, such as, for example,oil or water.

SUMMARY

According to one illustrative embodiment, a computer-implemented methodfor presenting a haptic hologram warning is provided. A computerreceives an indication that a first individual who needs supervision isapproaching a situation. The computer presents a haptic hologram to thefirst individual who needs supervision prior to the first individualreaching the situation. According to other illustrative embodiments, acomputer system and computer program product for presenting a haptichologram warning are provided.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial representation of a network of data processingsystems in which illustrative embodiments may be implemented;

FIG. 2 is a diagram of a data processing system in which illustrativeembodiments may be implemented;

FIG. 3 is a diagram illustrating an example of object warning systemcomponents in accordance with an illustrative embodiment;

FIG. 4 is a diagram illustrating an example of area analysis inaccordance with an illustrative embodiment;

FIG. 5 is a diagram illustrating an example of differentiation ofindividuals in accordance with an illustrative embodiment;

FIG. 6 is a diagram illustrating an example of object warning generationin accordance with an illustrative embodiment;

FIG. 7 is a diagram illustrating an example of ultrasound perception inaccordance with an illustrative embodiment;

FIGS. 8A-8B are a flowchart illustrating a process for determining ashape and size of a 3D holographic visual and haptic object warning inaccordance with an illustrative embodiment; and

FIG. 9 is a flowchart illustrating a process for presenting a haptichologram in accordance with an illustrative embodiment.

DETAILED DESCRIPTION

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

With reference now to the figures, and in particular, with reference toFIGS. 1-3, diagrams of data processing environments are provided inwhich illustrative embodiments may be implemented. It should beappreciated that FIGS. 1-3 are only meant as examples and are notintended to assert or imply any limitation with regard to theenvironments in which different embodiments may be implemented. Manymodifications to the depicted environments may be made.

FIG. 1 depicts a pictorial representation of a network of dataprocessing systems in which illustrative embodiments may be implemented.Network data processing system 100 is a network of computers, dataprocessing systems, image capturing devices, ultrasound transducerarrays, and other devices in which the illustrative embodiments may beimplemented. Network data processing system 100 contains network 102,which is the medium used to provide communications links between thecomputers, data processing systems, image capturing devices, ultrasoundtransducer arrays, and other devices connected together within networkdata processing system 100. Network 102 may include connections, suchas, for example, wire communication links, wireless communication links,and fiber optic cables.

In the depicted example, computer 104 and computer 106 connect tonetwork 102, along with storage 108. Computer 104 and computer 106 maybe, for example, computers with high-speed connections to network 102.In addition, computer 104 and computer 106 may be, for example, desktopor personal computers, laptop computers, handheld computers, such astablet computers, network computers, and the like. Also, it should benoted that computer 104 and computer 106 may represent other types ofdata processing systems, such as, for example, personal digitalassistants, smart phones, smart watches, gaming devices, set-top boxes,and the like.

Further, network data processing system 100 may be implemented in astructure, such as, for example, a residence, such as a house orapartment, an office space, a building, a set of two or more buildings,or the like. Furthermore, these structures may include a set of one ormore defined areas, such as area 110, area 112, and area 114. Area 110,area 112, and area 114 may represent different rooms or other types ofdefined or partitioned areas within a structure.

Each of area 110, area 112, and area 114 include image capturing device116 and ultrasound transducer array 118. Each image capturing device 116may be, for example, a video camera, a still picture camera, an infraredcamera, or any type or combination of devices capable of capturingimages. Each image capturing device 116 captures images of itscorresponding area, such as area 110, area 112, or area 114. The imagesmay be, for example, real time streaming images, series of still imagestaken at predetermined time intervals, or a combination of both.

Each of ultrasound transducer array 118 represents an array ofultrasound transducers that is configured to generate athree-dimensional (3D) holographic visual and haptic warningspatially-adjacent to an object, such as a hot stove, an exposed sharpcutting utensil, broken glass, and the like, within a correspondingarea, such as area 110, area 112, or area 114. A 3D holographic visualand haptic warning is a signal or alert, which can be seen and felt byan individual or person entering a space occupied by the 3D holographicvisual and haptic warning, associated with a monitored object within anarea.

Each of image capturing device 116 and each of ultrasound transducerarray 118 also connect to network 102. Image capturing device 116 andultrasound transducer array 118 are client devices of computer 104and/or computer 106. In other words, each of image capturing device 116sends captured image data of its corresponding area to computer 104and/or computer 106 for processing and analysis. In addition, each ofultrasound transducer array 118 receives control signals from computer104 and/or computer 106 to generate a 3D holographic visual and hapticobject warning based on the analysis of the received image data from animage capturing device. Furthermore, computer 104 and/or computer 106may provide information, such as software applications, programs, andupdates to image capturing device 116 and ultrasound transducer array118.

Storage 108 is a network storage device capable of storing any type ofdata in a structured format or an unstructured format. In addition,storage 108 may represent a plurality of network storage devices.Further, storage 108 may store identifiers for a plurality of imagecapturing devices and ultrasound transducer arrays, user profiles, listsof different defined areas, lists of objects to monitor in respectiveareas, lists of restricted objects within areas, lists of individualswith authorized access to restricted objects, lists of individuals withno access rights to restricted objects, and the like. Furthermore,storage unit 108 may store other types of data, such as authenticationor credential data that may include user names, passwords, access codes,and/or biometric data associated with users and system administrators,for example.

In addition, it should be noted that network data processing system 100may include any number of computers, data processing systems, imagecapturing devices, ultrasound transducer arrays, storage devices, andother devices not shown. Program code located in network data processingsystem 100 may be stored on a computer readable storage medium anddownloaded to a computer or other data processing device for use. Forexample, program code may be stored on a computer readable storagemedium on computer 104 and downloaded to one or more of image capturingdevice 116 over network 102 for use on image capturing device 116.

In the depicted example, network data processing system 100 may beimplemented as a number of different types of communication networks,such as, for example, an internet, an intranet, a local area network(LAN), a personal area network (PAN), a wireless fidelity (Wi-Fi)network, a peer-to-peer (P2P) network, and a wide area network (WAN).FIG. 1 is intended as an example only, and not as an architecturallimitation for the different illustrative embodiments.

With reference now to FIG. 2, a diagram of a data processing system isdepicted in accordance with an illustrative embodiment. Data processingsystem 200 is an example of a computer, such as computer 104 in FIG. 1,or other type of data processing system in which computer readableprogram code or instructions implementing processes of illustrativeembodiments may be located. In this illustrative example, dataprocessing system 200 includes communications fabric 202, which providescommunications between processor unit 204, memory 206, persistentstorage 208, communications unit 210, input/output (I/O) unit 212, anddisplay 214.

Processor unit 204 serves to execute instructions for softwareapplications and programs that may be loaded into memory 206. Processorunit 204 may be a set of one or more hardware processor devices or maybe a multi-processor core, depending on the particular implementation.Further, processor unit 204 may be implemented using one or moreheterogeneous processor systems, in which a main processor is presentwith secondary processors on a single chip. As another illustrativeexample, processor unit 204 may be a symmetric multi-processor systemcontaining multiple processors of the same type.

Memory 206 and persistent storage 208 are examples of storage devices216. A computer readable storage device is any piece of hardware that iscapable of storing information, such as, for example, withoutlimitation, data, computer readable program code in functional form,and/or other suitable information either on a transient basis and/or apersistent basis. Further, a computer readable storage device excludes apropagation medium. Memory 206, in these examples, may be, for example,a random-access memory, or any other suitable volatile or non-volatilestorage device. Persistent storage 208 may take various forms, dependingon the particular implementation. For example, persistent storage 208may contain one or more devices. For example, persistent storage 208 maybe a hard drive, a flash memory, a rewritable optical disk, a rewritablemagnetic tape, or some combination of the above. The media used bypersistent storage 208 may be removable. For example, a removable harddrive may be used for persistent storage 208.

In this example, persistent storage 208 stores object warning manager218, image data 220, and user profile 222. However, it should be notedthat even though object warning manager 218 is illustrated as residingin persistent storage 208, in an alternative illustrative embodimentobject warning manager 218 may be a separate component of dataprocessing system 200. For example, object warning manager 218 may be ahardware component coupled to communication fabric 202 or a combinationof hardware and software components.

Object warning manager 218 determines when to generate a 3D holographicvisual and haptic object warning and a size and shape of the objectwarning based on an analysis of received image data from an imagecapturing device corresponding to an area surrounding a set of one ormore monitored objects. In this example, object warning manager 218includes visual recognition analysis component 224 and ultrasoundtransducer array controller 226. However, it should be noted that objectwarning manager 218 may include fewer or more components thanillustrated. For example, visual recognition analysis component 224 andultrasound transducer array controller 226 may be combined into onecomponent and/or object warning manager 218 may include other componentsnot shown.

Image data 220 represents image data received from an image capturingdevice, such as image capturing device 116 in FIG. 1, which correspondsto area 228. Area 228 may be, for example, area 110 in FIG. 1. Area 228includes objects 230 and individuals 232. Objects 230 represent a set ofone or more objects in area 228. Individuals 232 represent a set of oneor more individuals in area 228. Image data 220 includes images ofobjects 230 and individuals 232.

Object warning manager 218 utilizes visual recognition analysiscomponent 224 to process and analyze image data 220 received from theimage capturing device to identify objects 230 and individuals 232 inarea 228, which corresponds to image data 220. Visual recognitionanalysis component 224 identifies a current state and status of objects230 in area 228, such as, for example, whether each heating elementobject on a stove top in area 228 is hot or cold. Further, visualrecognition analysis component 224 identifies each individual ofindividuals 232 in area 228, along with the age of each respectiveindividual, activity being performed by each respective individual, anda direction of movement of each respective individual in relation toobjects 230 being monitored in area 228.

Object warning manager 218 utilizes ultrasound transducer arraycontroller 226 to send control signals to an ultrasound transducerarray, such as ultrasound transducer array 118 in FIG. 1. Ultrasoundtransducer array controller 226 controls generation of a 3D holographicvisual and haptic warning by the ultrasound transducer array to alert anindividual to an object in area 228 that may present a potential dangerto the individual, for example. Object warning manager 218 directsultrasound transducer array controller 226 to generate the 3Dholographic visual and haptic warning using the ultrasound transducerarray based on the analysis of image data 220 received from visualrecognition analysis component 224.

User profile 222 contains information corresponding to a user of dataprocessing system 200. It should be noted that user profile 222 mayrepresent a plurality of different user profiles. The information inuser profile 222 may include, for example, restricted objects 234, timeperiods 236, and object access rules 238. Restricted objects 234represent a list of one or more objects, such as a birthday cake, thatthe user of data processing system 200 wants to limit access to in area228. Time periods 236 represent a set of one or more time periods thatthe user wants to limit access to restricted objects 234 during theperiods of time defined by time periods 236. For example, the user maynot want others to have access to a birthday cake until the date andtime of the birthday party. As another example, the user may not wantanyone, including the user, to have access to a kitchen refrigeratorbetween the hours of 8:00 p.m. and 6:00 a.m. the next morning for dietand/or health reasons.

Object access rules 238 represent rules for allowing or restrictingaccess to objects 230 in area 228 by individuals 232. Object accessrules 238 include authorized individuals 240 and restricted individuals242. Authorized individuals 240 represent a list of individuals that theuser authorizes to access restricted objects 234 or one or more ofobjects 230 in area 228. Restricted individuals 242 represent a list ofindividuals that the user wants to limit access to restricted objects234 or one or more of objects 230 in area 228. Object warning manager218 may utilize the information contained in user profile 222 todetermine whether to generate a 3D holographic visual and haptic objectwarning in addition to, or instead of, the image data analysisinformation received from visual recognition analysis component 224.

Communications unit 210, in this example, provides for communicationwith other computers, data processing systems, and devices via anetwork, such as network 102 in FIG. 1. Communications unit 210 mayprovide communications using both physical and wireless communicationslinks. The physical communications link may utilize, for example, awire, cable, universal serial bus, or any other physical technology toestablish a physical communications link for data processing system 200.The wireless communications link may utilize, for example, shortwave,high frequency, ultra-high frequency, microwave, Wi-Fi, Bluetooth®technology, global system for mobile communications (GSM), code divisionmultiple access (CDMA), second-generation (2G), third-generation (3G),fourth-generation (4G), 4G Long Term Evolution (LTE), LTE Advanced, orany other wireless communication technology or standard to establish awireless communications link for data processing system 200.

Input/output unit 212 allows for the input and output of data with otherdevices that may be connected to data processing system 200. Forexample, input/output unit 212 may provide a connection for user inputthrough a keypad and/or some other suitable input device. Display 214provides a mechanism to display information to a user and may includetouch screen capabilities to allow the user to make on-screen selectionsthrough user interfaces or input data, for example.

Instructions for the operating system, applications, and/or programs maybe located in storage devices 216, which are in communication withprocessor unit 204 through communications fabric 202. In thisillustrative example, the instructions are in a functional form onpersistent storage 208. These instructions may be loaded into memory 206for running by processor unit 204. The processes of the differentembodiments may be performed by processor unit 204 usingcomputer-implemented instructions, which may be located in a memory,such as memory 206. These program instructions are referred to asprogram code, computer usable program code, or computer readable programcode that may be read and run by a processor in processor unit 204. Theprogram instructions, in the different embodiments, may be embodied ondifferent physical computer readable storage devices, such as memory 206or persistent storage 208.

Program code 244 is located in a functional form on computer readablemedia 246 that is selectively removable and may be loaded onto ortransferred to data processing system 200 for running by processor unit204. Program code 244 and computer readable media 246 form computerprogram product 248. In one example, computer readable media 246 may becomputer readable storage media 250 or computer readable signal media252. Computer readable storage media 250 may include, for example, anoptical or magnetic disc that is inserted or placed into a drive orother device that is part of persistent storage 208 for transfer onto astorage device, such as a hard drive, that is part of persistent storage208. Computer readable storage media 250 also may take the form of apersistent storage, such as a hard drive, a thumb drive, or a flashmemory that is connected to data processing system 200. In someinstances, computer readable storage media 250 may not be removable fromdata processing system 200.

Alternatively, program code 244 may be transferred to data processingsystem 200 using computer readable signal media 252. Computer readablesignal media 252 may be, for example, a propagated data signalcontaining program code 244. For example, computer readable signal media252 may be an electro-magnetic signal, an optical signal, and/or anyother suitable type of signal. These signals may be transmitted overcommunication links, such as wireless communication links, an opticalfiber cable, a coaxial cable, a wire, and/or any other suitable type ofcommunications link. In other words, the communications link and/or theconnection may be physical or wireless in the illustrative examples. Thecomputer readable media also may take the form of non-tangible media,such as communication links or wireless transmissions containing theprogram code.

In some illustrative embodiments, program code 244 may be downloadedover a network to persistent storage 208 from another device or dataprocessing system through computer readable signal media 252 for usewithin data processing system 200. For instance, program code stored ina computer readable storage media in a data processing system may bedownloaded over a network from the data processing system to dataprocessing system 200. The data processing system providing program code244 may be a server computer, a client computer, or some other devicecapable of storing and transmitting program code 244.

The different components illustrated for data processing system 200 arenot meant to provide architectural limitations to the manner in whichdifferent embodiments may be implemented. The different illustrativeembodiments may be implemented in a data processing system includingcomponents in addition to, or in place of, those illustrated for dataprocessing system 200. Other components shown in FIG. 2 can be variedfrom the illustrative examples shown. The different embodiments may beimplemented using any hardware device or system capable of executingprogram code. As one example, data processing system 200 may includeorganic components integrated with inorganic components and/or may becomprised entirely of organic components excluding a human being. Forexample, a storage device may be comprised of an organic semiconductor.

As another example, a computer readable storage device in dataprocessing system 200 is any hardware apparatus that may store data.Memory 206, persistent storage 208, and computer readable storage media250 are examples of physical storage devices in a tangible form.

In another example, a bus system may be used to implement communicationsfabric 202 and may be comprised of one or more buses, such as a systembus or an input/output bus. Of course, the bus system may be implementedusing any suitable type of architecture that provides for a transfer ofdata between different components or devices attached to the bus system.Additionally, a communications unit may include one or more devices usedto transmit and receive data, such as a modem or a network adapter.Further, a memory may be, for example, memory 206 or a cache such asfound in an interface and memory controller hub that may be present incommunications fabric 202.

Illustrative embodiments generate 3D holographic visual and hapticobject warnings in mid-air in response to real time current conditionssurrounding a set of one or more monitored objects in a defined area.These 3D holographic object warnings have both a visual quality (i.e.,an individual can see them) and a haptic quality (i.e., an individualcan feel them). Through cognitive visual analysis of image datacorresponding to a given area surrounding the set of monitored objects,illustrative embodiments determine whether one or more visual and haptic3D holographic warnings are needed for one or more of the set ofmonitored objects in the given area. Illustrative embodiments generatethe 3D holographic visual and haptic warnings based on the analysis ofthe image data corresponding to the given area.

For example, illustrative embodiments may generate a visual and haptic3D holographic warning to discourage a child from touching a shape knifeon a kitchen surface, highlight where broken glass is located on afloor, discourage an adult from accessing a kitchen refrigerator atmidnight, or indicate that a birthday cake is not to be touched untilthe birthday party. However, it should be noted that illustrativeembodiments may selectively allow an adult who is cooking access to ahot stove without generating a visual and haptic 3D holographic warning,while generating a visual and haptic 3D holographic warning to a childapproaching the hot stove. Even though the generated 3D hologram doesnot physically obstruct an individual, the generated 3D hologram doesprovide both a visual and haptic 3D holographic warning to theindividual. As the individual enters an area and perceives the 3Dhologram by sight, touch, or both, illustrative embodiments provide theindividual with a warning to a nearby potentially dangerous object or auser-designated object with restricted access. Illustrative embodimentsprovide the visual warning, which an individual sees, via generation ofa 3D hologram. Illustrative embodiments provide the haptic warning,which the individual feels, as the individual enters the space occupiedby the 3D hologram.

Further, illustrative embodiments differentiate between differentindividuals and potentially dangerous situations and safe situations inan area. For example, a knife on a work surface in an area is typicallya safe situation for an adult, but is a potentially dangerous situationif the knife is within reach of a young child. Furthermore, illustrativeembodiments may dynamically modify or change 3D holographic visual andhaptic warnings to adapt to changing conditions in an area based oncognitive visual recognition analysis of real time streaming image dataof the area.

With reference now to FIG. 3, a diagram illustrating an example ofobject warning system components is depicted in accordance with anillustrative embodiment. Object warning system components 300 representhardware and software components for providing 3D holographic visual andhaptic object warnings. Object warning system components 300 may beimplemented in a network of data processing systems, such as networkdata processing system 100 in FIG. 1.

In this example, object warning system components 300 includes computer302, image capturing device 304, and ultrasound transducer array 306.Computer 302 may be, for example, computer 104 in FIG. 1 or dataprocessing system 200 in FIG. 2. Image capturing device 304 andultrasound transducer array 306 may be, for example, image capturingdevice 116 and ultrasound transducer array 118 in FIG. 1.

Computer 302 utilizes visual recognition analysis component 308 toreceive and analyze image data 310 from image capturing device 304.Visual recognition analysis component 308 may be, for example, visualrecognition analysis component 224 in FIG. 2. Visual recognitionanalysis component 308 provides cognitive interpretation of image data310 to detect situations where a warning needs to be generated betweenobject 312 and one or more individuals in the area surrounding object312.

Image capturing device 304 generates, for example, real time streamingimage data corresponding to the area surroundings object 312. In thisexample, object 312 is a stove. However, it should be noted that object312 may represent any type of monitored object.

Computer 302 utilizes ultrasound transducer array controller 314 tocontrol generation of 3D holographic visual and haptic warning 316 viaultrasound transducer array 306. Ultrasound transducer array 306 focusespatterns of ultrasound to shape 3D holographic visual and haptic warning316 in mid-air, which creates a warning that can be seen and felt by anindividual approaching object 312. It should be noted that object 312 iswithin an ultrasound pattern output range of ultrasound transducer array306. Ultrasound transducer array controller 314 varies the size andshape of 3D holographic visual and haptic warning 316 depending on thecurrent conditions and circumstances in the area surrounding object 312.

With reference now to FIG. 4, a diagram illustrating an example of areaanalysis is depicted in accordance with an illustrative embodiment. Areaanalysis 400 is performed by computer 402 based on analysis of imagedata 404 received from image capturing device 406, which corresponds toarea 408 surrounding object 410. Area 408 may represent, for example, akitchen in a house or apartment. In addition, a set of one or moreindividuals, such as individual 412, individual 414, and individual 416,may occupy area 408 surrounding object 410.

Image data 404 may be, for example, real time streaming video datacorresponding to area 408. Computer 402 utilizes visual recognitionanalysis component 418 to interpret image data 404 using cognitive deeplearning to understand the content and context of the images. Thus,visual recognition analysis component 418 can determine factors, such aswho is in a frame, the person's age, the person's activity, and theperson's direction of movement in area 408, and generate tags orclassifiers that identify area 408 and object 410 contained within imagedata 404.

In this example, image data 404 includes images of: object 410 (i.e., astove with all heating elements turned on and hot); individual 412having an age range of 35-44 and performing the activity of cooking;individual 414 having an age range of less than 12 and performing theactivity of running; and individual 416 having an age range of 35-44 andperforming the activity of using a handheld mobile device while walking.

With reference now to FIG. 5, a diagram illustrating an example ofdifferentiation of individuals is depicted in accordance with anillustrative embodiment. Differentiation of individuals 500 is performedby a computer, such as, for example, computer 402 in FIG. 4. Afterperforming area analysis 400 in FIG. 4 and classifying the objects andindividuals in the area, the computer determines whether to generate a3D holographic visual and haptic object warning based on the currentconditions and circumstances in the area.

Factors that the computer may utilize in making this determination mayinclude object warning pertinence or relevance. For example, thecomputer may rate a potentially dangerous object, such as, for example,a turned on stove with hot heating elements, higher as needing a warningas compared to a less dangerous object, such as, for example, a turnedoff stove with cold heating elements.

Age appropriateness also may be a factor. In other words, the age ofeach individual near an object may affect the need for a warning. Forexample, the computer may generate a warning for a child in the area,whereas the computer may not warn an adult in the same area.

Engaged activity also may be a factor. In other words, what each nearbyindividual is doing in the area also matters. For example, the computerdoes not warn individual 502 holding a saucepan from using the hotstove. However, the computer warns individual 504 not paying attentionwhile using a mobile phone in the area and individual 506 running in thearea.

Direction of movement also may be a factor. In other words, whether anindividual is moving toward object 508 or moving away from object 508.For example, if an individual, such as individual 506, is approachingobject 508, the computer may determine that the need for a warning ishigher.

The computer uses ultrasound transducer array 510 to generate 3Dholographic visual and haptic warning 512 between object 508 andindividuals 504 and 506 based on age and activities of individuals 504and 506. Ultrasound transducer array 510 may be, for example, ultrasoundtransducer array 306 in FIG. 3. In the example, ultrasound transducerarray 510 is an 8×4 array of ultrasound transducers. However, it shouldbe noted that ultrasound transducer array 510 is only meant as anexample and not as a limitation on different illustrative embodiments.In other words, ultrasound transducer array 510 may include any numberof ultrasound transducers and in any type of configuration to produce 3Dholographic visual and haptic warning 512.

With reference now to FIG. 6, a diagram illustrating an example ofobject warning generation is depicted in accordance with an illustrativeembodiment. Object warning generation 600 is performed by a computer,such as, for example, computer 402 in FIG. 4. After performingdifferentiation of individuals 500 in FIG. 5 and determining that anobject warning is needed, the computer determines a size and shape of 3Dholographic visual and haptic object warning 602 to generate. Thecomputer utilizes ultrasound transducer array 604 to generate 3Dholographic visual and haptic object warning 602.

The size and shape of 3D holographic visual and haptic object warning602 should conceal, or otherwise provide a visual and haptic warning, toa nearby individual who the computer determines is in need of a warning.If there are multiple individuals in the area, the size and shape of 3Dholographic visual and haptic object warning 602 can reflect that. Inthis example, individual 606 (i.e., running child) and individual 608(i.e., cooking adult) are shown in the vicinity of object 610 (i.e., ahot stove). The computer determines that individual 608 needs access toobject 610, whereas individual 606 needs a warning. The size and shapeof 3D holographic visual and haptic object warning 602 reflects thissituation. Further, the computer may modify the intensity of the hapticfeel of 3D holographic visual and haptic object warning 602. Forexample, the computer may increase or decrease the sensation individual606 feels based on the computer determining the level or degree ofdanger posed to individual 606 by object 610.

In this example, the computer visually and haptically warns individual606 of potentially dangerous object 610, but does not warn individual608. In addition, the computer continuously analyzes new streaming videocontent as it is received. As a result, the computer may change ormodify 3D holographic visual and haptic object warning 602 based onchanging conditions and circumstances in the area surrounding object610. For example, the computer may continue to modify the size and shapeof 3D holographic visual and haptic object warning 602 as individualscontinue to move around object 610. Further, as a state of object 610changes (e.g., the stove is now turned off and heating elements arecool), the computer may remove or stop generation of 3D holographicvisual and haptic object warning 602. As the computer detects newobjects, individuals, conditions, circumstances, and/or potentialdangers, the computer may add new 3D holographic visual and hapticobject warnings in the area.

With reference now to FIG. 7, a diagram illustrating an example ofultrasound perception is depicted in accordance with an illustrativeembodiment. Ultrasound perception 700 illustrates how ultrasoundtransducers 702 generate ultrasound pattern 704. Individual 706perceives ultrasound pattern 704 as individual 706 enters a spaceoccupied by ultrasound pattern 704. However, it should be noted thateven though a hand of individual 706 is shown in this example asperceiving ultrasound patter 704, any portion of individual 706's bodymay perceive ultrasound pattern 704.

With reference now to FIGS. 8A-8B, a flowchart illustrating a processfor determining a shape and size of a 3D holographic visual and hapticobject warning is shown in accordance with an illustrative embodiment.The process shown in FIGS. 8A-8B may be implemented in a computer, suchas, for example, computer 104 in FIG. 1, data processing system 200 inFIG. 2, or computer 402 in FIG. 4.

The process begins when the computer receives image data of an areasurrounding an object from an image capturing device (step 802). Theimage data of the area surrounding the object from the image capturingdevice may be, for example, image data 404 of area 408 surroundingobject 410 from image capturing device 406. Also, it should be notedthat the image data may be real time continuously streaming image dataof the area. In addition, the computer performs an analysis of the imagedata using a visual recognition analysis component of the computer, suchas visual recognition analysis component 418 in FIG. 4 (step 804).

Further, the computer identifies the object, a current status of theobject, a set of one or more individuals in the area surrounding theobject, age of each individual in the set, activity of each individualin the set, and direction of movement of each individual in the setbased on the analysis of the image data (step 806). The current statusof the object may be, for example, that the object is in a turned-onstate and is in an active mode of operation, such as increasingtemperature in heating elements or activating paper shredding elements.The set of individuals in the area surrounding the object may be, forexample, individuals 412-416 in FIG. 4. The direction of movement ofeach individual may be, for example, no movement (i.e., stationary),movement toward the object, movement away from the object, movementparallel to the object, and the like.

Furthermore, the computer makes a determination as to whether generatinga 3D holographic visual and haptic warning is pertinent to the objectbased on the current status of the object and the age, activity, anddirection of movement of each individual in the set of individualswithin the area surrounding the object (step 808). If the computerdetermines that generating a 3D holographic visual and haptic warning isnot pertinent to the object based on the current status of the objectand the age, activity, and direction of movement of each individual inthe set of individuals within the area surrounding the object, no outputof step 808, then the process returns to step 802 where the computercontinues to receive image data of the area. If the computer determinesthat generating a 3D holographic visual and haptic warning is pertinentto the object based on the current status of the object and the age,activity, and direction of movement of each individual in the set ofindividuals within the area surrounding the object, yes output of step808, then the computer determines a shape and a size of the 3Dholographic visual and haptic warning to generate between the object andone or more individuals in the set of individuals based on the currentstatus of the object and the age, activity, and direction of movement ofeach individual within the area surrounding the object (step 810).

Afterward, the computer generates the 3D holographic visual and hapticwarning between the object and the one or more individuals in the set ofindividuals in accordance with the determined size and shape (step 812).Moreover, the computer makes a determination as to whether generatingthe 3D holographic visual and haptic warning is still pertinent to theobject based on analyzing new image data of the area surrounding theobject (step 814). If the computer determines that generating the 3Dholographic visual and haptic warning is still pertinent to the objectbased on analyzing the new image data of the area surrounding theobject, yes output of step 814, then the process returns to step 810where the computer continues to determine the size and shape of the 3Dholographic visual and haptic warning to generate.

If the computer determines that generating the 3D holographic visual andhaptic warning is not pertinent to the object based on analyzing the newimage data of the area surrounding the object, no output of step 814,then the computer stops generation of the 3D holographic visual andhaptic warning (step 816). Thereafter, the process returns to step 802where the computer continues to receive image data of the area.

With reference now to FIG. 9, a flowchart illustrating a process forpresenting a haptic hologram is shown in accordance with an illustrativeembodiment. The process shown in FIG. 9 may be implemented in acomputer, such as, for example, computer 104 in FIG. 1, data processingsystem 200 in FIG. 2, or computer 402 in FIG. 4.

The process begins when the computer receives an indication that eitheran individual who needs supervision is approaching a potentiallydangerous situation or an individual who does not need supervision isapproaching the potentially dangerous situation (step 902). Theindividual who needs supervision may be, for example, individual 606 inFIG. 6. The individual who does not need supervision may be, forexample, individual 608 in FIG. 6. The dangerous situation may be, forexample, object 610 in FIG. 6 being an electric stove with the heatingelements turned on and hot.

The computer presents a haptic hologram to the individual who needssupervision prior to that individual reaching the potentially dangeroussituation while refraining from presentation of the haptic hologram tothe individual who does not need supervision (step 904). The haptichologram may be, for example, 3D holographic visual and haptic warning602 in FIG. 6. Thereafter, the process terminates.

Thus, illustrative embodiments of the present invention provide acomputer-implemented method, computer system, and computer programproduct for generating a 3D holographic visual and haptic warning basedon visual recognition analysis of streaming image data corresponding toan area surrounding a set of one or more monitored objects. Thedescriptions of the various embodiments of the present invention havebeen presented for purposes of illustration, but are not intended to beexhaustive or limited to the embodiments disclosed. Many modificationsand variations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A computer-implemented method for presenting ahaptic hologram warning, the computer-implemented method comprising:receiving, by a computer, an indication that a first individual whoneeds supervision is approaching a situation; and presenting, by thecomputer, a haptic hologram to the first individual who needssupervision prior to the first individual reaching the situation.
 2. Thecomputer-implemented method of claim 1 further comprising: receiving, bythe computer, an indication that a second individual who does not needsupervision is approaching the situation; and refraining, by thecomputer, from presentation of the haptic hologram to the secondindividual who does not need supervision.
 3. The computer-implementedmethod of claim 1, wherein the situation is one of a group consisting ofa potentially dangerous object and a user-designated object withrestricted access.
 4. The computer-implemented method of claim 1 furthercomprising: receiving, by the computer, image data of an areasurrounding an object from an image capturing device; and performing, bythe computer, an analysis of the image data using a visual recognitionanalysis component of the computer.
 5. The computer-implemented methodof claim 4, wherein the image data is real time streaming image data ofthe area surrounding the object.
 6. The computer-implemented method ofclaim 4 further comprising: identifying, by the computer, the object, acurrent status of the object, a set of individuals in the areasurrounding the object, age of each individual in the set, activity ofeach individual in the set, and direction of movement of each individualin the set based on the analysis of the image data.
 7. Thecomputer-implemented method of claim 6 further comprising: determining,by the computer, whether generating a three-dimensional (3D) holographicvisual and haptic warning is pertinent to the object based on thecurrent status of the object and the age, activity, and direction ofmovement of each individual in the set of individuals within the areasurrounding the object; and responsive to the computer determining thatgenerating a 3D holographic visual and haptic warning is pertinent tothe object based on the current status of the object and the age,activity, and direction of movement of each individual in the set ofindividuals within the area surrounding the object, determining, by thecomputer, a size and a shape of the 3D holographic visual and hapticwarning to generate between the object and one or more individuals inthe set of individuals.
 8. The computer-implemented method of claim 7further comprising: generating, by the computer, the 3D holographicvisual and haptic warning between the object and the one or moreindividuals in the set of individuals in accordance with the determinedsize and shape using an ultrasound transducer array in the area.
 9. Thecomputer-implemented method of claim 8 further comprising: determining,by the computer, whether the generating of the 3D holographic visual andhaptic warning is still pertinent to the object based on analyzing newimage data of the area surrounding the object; and responsive to thecomputer determining that the generating of the 3D holographic visualand haptic warning is not pertinent to the object based on the analyzingof the new image data of the area surrounding the object, stopping, bythe computer, generation of the 3D holographic visual and hapticwarning.
 10. The computer-implemented method of claim 1, wherein thecomputer modifies an intensity of the haptic hologram based on thecomputer determining a level of danger posed to the first individual bythe situation.
 11. A computer system for presenting a haptic hologramwarning, the computer system comprising: a bus system; a storage deviceconnected to the bus system, wherein the storage device stores programinstructions; and a processor connected to the bus system, wherein theprocessor executes the program instructions to: receive an indicationthat a first individual who needs supervision is approaching asituation; and present a haptic hologram to the first individual whoneeds supervision prior to the first individual reaching the situation.12. The computer system of claim 11, wherein the processor furtherexecutes the program instructions to: receive an indication that asecond individual who does not need supervision is approaching thesituation; and refrain from presentation of the haptic hologram to thesecond individual who does not need supervision.
 13. A computer programproduct for presenting a haptic hologram warning, the computer programproduct comprising a computer readable storage medium having programinstructions embodied therewith, the program instructions executable bya computer to cause the computer to perform a method comprising:receiving, by the computer, an indication that a first individual whoneeds supervision is approaching a situation; and presenting, by thecomputer, a haptic hologram to the first individual who needssupervision prior to the first individual reaching the situation. 14.The computer program product of claim 13 further comprising: receiving,by the computer, an indication that a second individual who does notneed supervision is approaching the situation; and refraining, by thecomputer, from presentation of the haptic hologram to the secondindividual who does not need supervision.
 15. The computer programproduct of claim 13, wherein the situation is one of a group consistingof a potentially dangerous object and a user-designated object withrestricted access.
 16. The computer program product of claim 13 furthercomprising: receiving, by the computer, image data of an areasurrounding an object from an image capturing device; and performing, bythe computer, an analysis of the image data using a visual recognitionanalysis component of the computer.
 17. The computer program product ofclaim 16, wherein the image data is real time streaming image data ofthe area surrounding the object.
 18. The computer program product ofclaim 16 further comprising: identifying, by the computer, the object, acurrent status of the object, a set of individuals in the areasurrounding the object, age of each individual in the set, activity ofeach individual in the set, and direction of movement of each individualin the set based on the analysis of the image data.
 19. The computerprogram product of claim 18 further comprising: determining, by thecomputer, whether generating a three-dimensional (3D) holographic visualand haptic warning is pertinent to the object based on the currentstatus of the object and the age, activity, and direction of movement ofeach individual in the set of individuals within the area surroundingthe object; and responsive to the computer determining that generating a3D holographic visual and haptic warning is pertinent to the objectbased on the current status of the object and the age, activity, anddirection of movement of each individual in the set of individualswithin the area surrounding the object, determining, by the computer, asize and a shape of the 3D holographic visual and haptic warning togenerate between the object and one or more individuals in the set ofindividuals.
 20. The computer program product of claim 19 furthercomprising: generating, by the computer, the 3D holographic visual andhaptic warning between the object and the one or more individuals in theset of individuals in accordance with the determined size and shapeusing an ultrasound transducer array in the area.